Immersive video environment using near-infrared video compositing

ABSTRACT

Various methods and systems are disclosed for near-infrared video compositing techniques and an associated immersive video environment. In an example, a video environment includes: a visible light camera and an infrared detection camera arranged to capture video from a performance area; a visible light source and infrared light source arranged to emit visible light onto the performance area; a display source to provide video output; and a display screen arranged in the performance area between the camera system and the infrared light source. The display screen is further arranged to reflect visible light originating from the display source, while permitting infrared and visible light from the performance area to reach the cameras. In a further example, the system includes a backdrop integrating the infrared light source, as infrared light from the infrared light source passes through the backdrop into the performance area.

RELATED APPLICATIONS

The subject matter of the present application is related to U.S. patentapplication Ser. No. 15/713,057 to Lovemelt et al., titled NEAR-INFRAREDVIDEO COMPOSITING, and filed Sep. 22, 2017, the contents of which areincorporated by reference herein in their entirety.

TECHNICAL FIELD

Embodiments described herein generally relate to video processing andvisual effect techniques and environments, and in particular, but not byway of limitation, to systems, methods, and accompanying embodimentsinvolving compositing video from visible light and near-infraredcameras, and providing outputs from such composited video in aninteractive environment.

BACKGROUND

A variety of techniques are currently used for the creation andmanipulation of video post-production effects. One common techniqueinvolves the use of chroma key compositing, which composites (layers)two or more images or video streams based on color hues. A well-knownexample of this type of video effect is used in many television newsproductions, which superimpose a human presenter who is captured infront of a chroma key screen (e.g., a “green screen”) over acomputer-generated weather map or other computer-generated content.Another common technique for compositing involves the use of visiblebackground subtraction, which is performed after calculating the colordistance between a known background image and an input video stream.With the background content removed, the video stream can then becomposited with other video content.

These and similar video compositing techniques involve the use andprocessing of visible information to identify the boundaries betweendesired and undesired video content. Unfortunately, the effects fromsuch composting techniques may be distorted or unreliable during unevenlighting or incorrect camera exposure conditions. These compositingtechniques are also imprecise and often experience difficulty intracking movement. As a result, chroma key compositing and backgroundsubtraction are unable to be cleanly used in many low-light and realworld settings.

Limited techniques have been proposed for the use of infrared videocompositing to segment video content without being restricted by thelimitations of visible light as described above. For instance, a 1960paper published by Zoli Vidor, “An Infrared Self-Matting Process”,discusses the use of compositing through an infrared traveling matteprocess, provided from visible and infrared light exposures captured onseparate sets of film. The applicability of the Vidor technique,however, is limited due to the use of a specialized camera and thecomplexity of film-based processing. A more recent example, a 2002 paperpublished by Paul Debevec et al., “A Lighting Reproduction Approach toLive-Action Compositing”, discusses the use of live-action matting andcompositing with digital video with use of near-infrared light. However,the Debevec paper emphasizes the use of a near-infrared camera within aspecialized light stage for the purpose of replicating lighting specialeffects from complex motion picture scenes. As a result, infrared videocompositing has only been applied in limited settings, such as incomplex and artificial video capture stages or research environments.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notof limitation, in the figures of the accompanying drawings, in which:

FIGS. 1A-1C illustrate side and front views of an environment forcapturing video of a subject using a near-infrared video compositingcamera system, according to an example;

FIGS. 2A-2B illustrate exterior views of an interactive unit forcapturing video of a subject, according to an example;

FIG. 2C illustrates a top perspective view of the interactive unit forcapturing video of a subject, according to an example;

FIGS. 2D-2E illustrate interior views of the interactive unit forcapturing video of a subject, according to an example;

FIGS. 3A-3C illustrate front and perspective views of the near-infraredvideo compositing camera system used with in a video telepromptersystem, according to an example;

FIGS. 4A-4C illustrate top and perspective views of the near-infraredvideo compositing camera system used with in a video projection system,according to an example;

FIGS. 5A-5B illustrate side views of infrared and visible light sourcesfor use with the near-infrared video compositing camera system,according to an example;

FIG. 6 illustrates video capture and output from the near-infrared videocompositing camera system used with in a video teleprompter system,according to an example;

FIG. 7 illustrates an overview of a visible and near-infrared videoprocessing sequence for video compositing, according to an example;

FIG. 8 is a flowchart of an example technique for generating a matte andperforming video compositing with the matte via an electronic processingsystem, according to various examples;

FIG. 9 is a flowchart of an example method for video compositing basedon infrared video, according to various examples;

FIG. 10 is a block diagram of example processing components andsubsystems operable for video compositing based on near-infrared video,according to various examples; and

FIG. 11 is a block diagram of a machine in the example form of anelectronic computing system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of some example embodiments. It will be evident, however,to one skilled in the art that aspects of the present disclosure may bepracticed without these specific details.

In various examples described herein, an interactive video processingsystem utilizing features of infrared video compositing is arranged tocreate a matte. In an example, the interactive video processing systemincludes use of a dual camera system, including a near-infrared cameraand a visible light camera arrangement to provide dual video feeds of acommon field of view to a digital video processing system. Accompanyingprocessing methods are used to extract a matte from the dual videofeeds, composite the matte with other video content, and select andapply additional video post-processing effects (as applicable).

In an example, the interactive video processing system is utilized withaspects of a video input/output display system that provides immediatefeedback to a human user in the form of a real-time video display. In anexample, the video input/output display system is embodied withinfeatures of a specialized teleprompter that is viewable by a human user.The specialized teleprompter may include the dual camera system arrangedbehind a teleprompter display to capture video from a subject area whilealso projecting the real-time video display to the subject area. Inanother example, the video input/output apparatus is embodied withfeatures of a specialized projection screen. The specialized projectionscreen may include a reflective material that allows a video projectionof the human user to be displayed to the subject area, while allowingthe dual camera system to capture light from the subject area throughthe specialized projection screen.

In an example, the interactive video processing system is utilized withaspects of an interactive video stage, embodied by a video booth, videocapture unit, or similar performance area. This interactive video stagemay be designed to provide users with a unique video experience that cancapture a performance, display the performance, and produce digitalrecorded content, in real-time. As discussed herein, this interactivevideo stage may be used with features of the interactive videoprocessing system (e.g., a computing system controlling inputs andoutputs with the specialized teleprompter or specialized projectionscreen) to provide an intuitive environment for high speed video captureand real-time video manipulation, while offering robust lightingcapabilities in a dynamic compositing framework. In further examples,the interactive video processing system may control aspects ofadditional post-processing and environment special effects relating tovideo, sound, light, smoke, wind, or other aspects. Accordingly, theinteractive video processing system may offer robust capacities suitablefor advanced video art installations within a variety of environmentssuch as museums, galleries, clubs, or experiential marketing locations.

As discussed above, existing approaches for segmentation of a human userand real-world objects in video are based on chroma keying, backgroundsubtraction, and like post-processing techniques. These techniques areoften not suitable for use in real-world settings beyond complex orexpensive video stages. The interactive video stage configurationdiscussed herein provides an alternative to traditional green screenvideo stages, to allow a near-infrared video compositing camera toaccurately capture a matte in darkness or dynamic lighting situations.The interactive video stage configuration may be embodied by anenclosure or structure that hosts the near-infrared video compositingcamera system and allows real-time playback and output on a displayscreen. Further, the near-infrared video compositing camera system maybe located within the interactive video stage configuration in such amanner to allow the display and capture of video from a common area,thus providing a more intuitive and easy to use environment than manyconventional uses of video monitors and video booths where the displayscreen and cameras are displaced from one another.

As discussed herein, the present systems and techniques for infraredvideo compositing, human and object segmentation, and video capture andprocessing are applicable to a variety of professional and informal (andpublic and private) environments. As such, the present systems andtechniques are not limited to the specific interactive video stage ordual camera configurations discussed herein. Further, it will beapparent that the many of the illustrated and described arrangements ofthe components, such as the camera systems, display screens, lightingsources, and special effects sources (e.g., surround or directionalsounds, smoke, lasers, strobe lights, wind machines, fans) describedherein may be modified, operated, and rearranged without departing fromthe inventive concepts for the use of infrared video compositing.

As also discussed herein, numerous references are made to “visible”light and “infrared” or “near-infrared” light. References to “visible”light are intended to refer to human-visible wavelengths of light (e.g.,within the wavelength range between 400-700 nanometers (nm)), which arecaptured by visible light sensors of an appropriately configured digitalcamera. References to “infrared” and “IR” are intended to refer tohuman-invisible wavelengths of light (e.g., within the wavelength rangebetween 700 nm to 1 millimeters (mm)) extending beyond the visible lightrange, which are captured by infrared light sensors of an appropriatelyconfigured camera. Specifically, such infrared light sensors aretypically capable of identifying “near-infrared” light, in a smallersub-division of the infrared light band, such as located between 700nm-1000 nm. The following references that are to “infrared”, “IR”,“near-infrared” and “NIR” refer interchangeably to such near-infraredlight in the infrared wavelength range that is perceivable by infraredlight sensors. In an example, references that are made herein to“visible” may refer to light in the visible light range that isperceivable by visible light sensors. In an example, a visible lightdetector and an infrared light detector can detect overlappingfrequencies of light. Similarly, references to “visible light” caninclude a spectrum of light that can include light extending into theinfrared range. For example, a visible light source may emit visible and(possibly incidentally) infrared light, such as near red infrared light.In some examples, an infrared light source does not emit visible light.

FIGS. 1A-1C illustrate side and front views of an example environmentfor capturing video of a subject using a NIR video compositing camerasystem. Specifically, FIG. 1A depicts a perspective view 100A of a videocapture environment, having one or more objects (specifically, a humanuser 110) located within a field of view of a set of cameras hosted in aspecialized teleprompter video display system 200. A specificconfiguration and layout of a structure used to host the video captureenvironment is further detailed below with reference to FIGS. 2A-2E.

The teleprompter video display system 200 includes a dual camera system300 arranged to capture IR and visible light video from the field ofview. The dual camera system 300 is used to produce respective streamsof IR and visible light video, as discussed herein, for creating a matteof the human user 110 and any other objects (including other humanusers) within the field of view. A specific configuration and layout ofthe teleprompter video display system 200 is further detailed below withreference to FIGS. 3A-3C. Additional description of the arrangement ofthe camera system 300 is provided in U.S. patent application Ser. No.15/713,057 to Hal Lovemelt et al., filed Sep. 22, 2017, and titledNEAR-INFRARED VIDEO COMPOSITING, which is incorporated by reference inits entirety. In another example, a video projection system 400 may besubstituted for the teleprompter video display system 200 as shown anddescribed with reference to FIGS. 4A-4C. As will be apparent from thefollowing examples, a variety of projection and camera capture mediumsand arrangements may be utilized to capture the field of view with useof the dual camera system 300.

As depicted in FIGS. 1A-1C, the human user 110 is located between theteleprompter video display system 200 and a backdrop 120, and the humanuser 110 is able to move into, within, and out of a field of viewbetween the backdrop 120 and the teleprompter video display system 200.A set of one or more lights 130 are located in the environment toilluminate visible light (e.g., white light, or colored lights) on thefield of view to be captured by the cameras. For example, the lights 130may be suspended within a light stage or other structure (not shown), toallow visible light to be illuminated on human user 110 and otherobjects. Only one of the lights 130 in FIGS. 1A-1C is labeled forsimplicity; the number, position, orientation of the lights 130 may varysignificantly depending on the size of the field of view, theenvironment, the stage or structure, the shape and size of the lights,and like factors. Further, the lights 130 may include various electricor electronic controls to incorporate dynamic lighting angles, colorpalettes, or light effects.

The backdrop 120 may provide a boundary or other physical definition forone or more areas of the field of view. In an example, the surface 122of the backdrop 120 that is visible in the field of view by the dualcamera system 300 may provide a dark or other high-contrast background(e.g., black, or dark gray) to absorb visible light. In a specificexample, the backdrop is a black masking fabric that appears a solidblack to a RGB camera, but which allows IR light to be shined throughthe fabric to provide IR glow diffusion that is detectable by a NIRcamera.

In an example, the backdrop 120 is positioned relative to an IR lightsource to allow IR light to be emitted into the field of view through avisible surface 122 of the backdrop 120. In a further example, thesurface 122 of the backdrop 120 is provided from a fabric material thatpermits IR light to pass through (from the backdrop 120 towards theteleprompter video display system 200 and the dual camera system 300),while the material of the surface 122 absorbs visible light, orsignificantly reduces the reflection of visible light, from the lights130 or other lighting sources.

As discussed in the examples below, the dual camera system 300 and anaccompanying video processing system may be adapted to detect objects inthe field of view using rear IR illumination that establishes asilhouette of the objects in the field of view. In other examples notdepicted in the drawings, the backdrop 120 may be alternatively oradditionally illuminated by projecting light from within the field ofview towards the backdrop 120. Further, the techniques discussed hereinmay be adapted for front or side IR illumination, or other variations toIR matting.

In an example, the teleprompter video display system 200 includes ateleprompter housing 210 and a display source 220, with the displaysource 220 arranged to output a projection onto a two-way display screenwithin the housing 210 (e.g., to project onto a two-way display screen250, depicted in the perspective of FIG. 1C). The two-way display screen250 is positioned within the housing 210 to reflect light from thedisplay source 220 projected at a first angle, while allowing lightentering the housing 210 from a second angle to reach the cameras (e.g.,to be visible to the dual camera system 300). In an example, the displaysource 220 may be provided from a monitor, projector, or otherlight-emitting source, and is positioned on a rear end of theteleprompter video display system 200. The teleprompter video displaysystem 200 may utilizes one or more mirrors to reflect and project thelight emitted from the display source 220 onto the two-way displayscreen 250. In another example, the display source 220 may be positioneddirectly under the two-way display screen, to directly project the lightonto the two-way display screen 250. A variety of other arrangements andmaterials utilized by existing forms of teleprompters and video monitorsmay be integrated or adapted into the teleprompter video display system200.

The two-way display screen 250 allows light to be captured by the camerasystem 300 camera, as received from the field of view through an opening240 (e.g., an opening of the teleprompter housing 210), while providinga display that is perceivable to an observer (e.g., human user 110). Inan example, the dual camera system 300 is positioned on a rear platform230 of the teleprompter video display system 200 with use of an elevatedcamera platform 235. The positioning of the dual camera system 300relative to the opening 240 allows the dual camera system 300 to captureIR and visible light from the field of view (and the objects such as thehuman user 110 that are illuminated or silhouetted within the field ofview). Other forms of platforms, tripods, mounts, and the like may beused to position and adjust the dual camera system 300 or theteleprompter video display system 200.

FIG. 1B depicts a front view 100B of the video capture environment,showing the perspective of the dual camera system 300 from behind thedual camera system 300 and the teleprompter video display system 200(e.g., in the same direction that the IR and visible light camerascapture the field of view). Accordingly, the dual camera system 300 isarranged to capture video of the field of view, from light receivedwithin the opening 240 through the two-way display screen 250. A moredetailed perspective illustration of how the camera output is capturedvia the dual camera system 300 and output via the two-way display screen250 is discussed below and depicted with reference to FIG. 6.

FIG. 1C depicts another perspective view 100C of the video captureenvironment, showing the human user 110 located within the field of viewof the teleprompter video display system 200. In this perspective view100C, the two-way display screen 250 is visible. In an example, thetwo-way display screen 250 is arranged reflect a video output projectedfrom the display source 220 through a mirror or a series of mirrors (notshown). The two-way display screen 250 is arranged relative to the dualcamera system 300 to allow IR light (e.g., emitted from the backdrop120) and visible light (e.g., emitted from the lights 130) from thefield of view, to travel through the two-way display screen 250, andthrough the opening 240, to reach the dual camera system 300. In anexample, the light may reach a hot mirror of the dual camera system 300,used to split IR and visible light. Further illustrations of the hotmirror of the dual camera system 300, and the positioning of the IR andvisible light cameras relative to the opening 240, are described below.

FIG. 1C also depicts a rear side of the backdrop 120, outside of thefield of view. Here, the rear side of the backdrop 120 is structured tohost a series of IR light emitters 124, such as with an array of lightbars emitting NIR wavelengths through the backdrop 120, towards thefield of view. These IR light emitters 124 may be used to provide abacklit environment of objects in the field of view, as the IR lightemitters 124 emit IR light in the direction of the dual camera system300. In a specific example, the backdrop 120 includes a plasticdiffusion unit affixed to each strip of LEDs within the array of IRlight emitters 124; additionally, approximately 6 inches from the LEDs,a dual-vision projection surface may be disposed within the backdrop 120to act as an infrared diffusion layer. As a result, a consistent anduniform glow may be emitted through the backdrop 120 from the variousemitters 124.

Based on the configurations depicted in the present disclosure, visible(color) lighting within the environment is positioned to create thewidest range of lighting styles and looks for video portraiture. In anexample, each light of the lights 130 is arranged equidistant from theperformance center point, and the lights beams may be spread to anappropriate width to encompass a range of human heights. Further, the IRlighting may be positioned in an array at an appropriate distance from adiffusion surface (e.g., which has been layered with a matte soft blackfabric) of the backdrop 120, to create fully glowing surface of thebackdrop in IR, while also absorbing visible light. With such consistentbackdrop IR illumination, the video processing system is able to createa very efficient and accurate matte—even allowing for capturing aspectsof transparency in a performer's hair or wearables. A more detailedperspective illustration of how the visible and IR light is emitted atthe objects within the field of view is discussed below and depictedwith reference to FIG. 5B. Additional description of the arrangement ofthe camera system 300 is provided in U.S. patent application Ser. No.15/713,057 to Hal Lovemelt et al., filed Sep. 22, 2017, and titledNEAR-INFRARED VIDEO COMPOSITING, which is incorporated by reference inits entirety.

FIGS. 2A-2B illustrate exterior views of an example interactive stagestructure 150 adapted for capturing and displaying video of a subjectwith hosting of the teleprompter video display system 200, dual camerasystem 300, and the other input, output, and processing componentsdescribed herein. Specifically, FIG. 2A and FIG. 2B each provideperspective views of an exterior of the interactive stage structure 150,with the structure being sized to host a human user and define acontrolled area for the camera field of view.

As shown in FIG. 2A, the interactive stage structure 150 includes a setof walls 164 (which may include curtains or other material to block outlight, e.g., not limited to a rigid structure) attached via a frame 160and posts 162. For simplicity, only one of the walls 164, posts 162, andframe sides is labeled. FIG. 2A illustrates the placement of the frame160 and posts 162, surrounding the backdrop 120. The backdrop 120accordingly establishes one of four sides of the interactive stagestructure 150, whereas the walls 164 establish the remaining sides. FIG.2B illustrates a perspective view that is opposite of the view of FIG.2A, with multiple of the walls 164 visible.

In an example, the walls 164 include one or more openings (e.g., a dooror a slot in a curtain) on one or more sides, to allow a human user orother objects to enter the interior chamber defined by the interactivestage structure 150. Also in an example, an exterior surface 166 of thewalls 164 allows the attachment or coupling of materials. For instance,the walls 164 may be “wrappable” with various forms of skins orwrappings, to allow advertisements, logos, or other branding or designsto be applied to the structure 150 (including in the form of variousexterior “shell” structures). A variety of attachment mechanisms andmaterials may be used to couple such skins or wrappings to the interioror exterior of the structure 150. Further, although the walls 164 arenot shown as extending to a floor, other variations to the structure 150may provide a completely closed (and user-immersed) environment.

Also as an example, a cover over the structure 150 may be arranged toact as a roof on the structure, to enclose the structure and separatethe interior chamber from an outside environment. In another example,the structure shown in FIG. 2A may be further enclosed within a wrap orother structure to curtail ambient light. For example, the structure inFIG. 2A may not include walls, but may be situated within a larger lightblocking structure. In still other examples, the shape of the structuremay be adapted into a dome, sphere, pod, cage, or other formats. Also asan example, the structure 150 may be sized to host a small number ofpersons (e.g., 1-4 persons) and small objects and props; in otherexamples, the structure may be sized to host a larger group of persons(e.g., between 8-12 persons) or larger objects or props.

In an example, various user interactive features may be presented orhosted on the exterior of the interactive stage structure 150 for userengagement or information display purposes. For instance, a displayscreen 170 may be mounted on an exterior location (e.g., coupled to oneof the posts 162), with the display screen providing a real-time,third-person perspective display of aspects of the interior of theenvironment, which may include a virtual background composited withvideo. In an example, the display screen 170 may output a real-timedisplay of the video being captured from within the interactive stagestructure 150. In another example, the display screen 170 may outputinstructions or other information related to the capture of video fromwithin the interactive stage structure 150 or use of the dual camerasystem 300.

In still another example, the display screen 170 and accompanying inputdevices (e.g., a touch screen within the display screen 170, avoice-interactive interface, and the like) may provide user-interactivefeatures to collect information on the user, such as desired videobackgrounds or special effects, preferences, and environment controls orthemes. The user-interactive features provided may also integrate withinternet or app-based services, such as to allow a user to controlspecial effects via input on the user's smartphone, and to playback ordownload a video clip produced from the video environment. For instance,a user may interact with the display screen 170 to choose and configurethe experience to occur within the video environment; the user may alsoenter contact information to electronically obtain content from theperformance within the video environment.

In still another example, the display screen 170 (or other externaldevices) may be used by staff or an administrator to control and directa user's experience within the environment. For instance, a director maymonitor a user's performance within the environment, control aspects ofvideo, lighting, audio, and effects, and provide direction and guidanceto the user via communications from outside the environment. Otheraspects of electronic communication, control, and display devices thatare not depicted may be used by a director or other trained staff.

FIG. 2C illustrates a top perspective view of the interactive stagestructure 150, defining an interior space for capturing video of asubject. As shown, the frame 160, posts 162, and walls 164 define aninterior chamber in which the human user 110 can move within (and intoand out of). As shown, the field of view that exists between the dualcamera system 300 (and the teleprompter video display system 200) andthe backdrop 120 is illuminated by the various lights 130. FIG. 2Cfurther illustrates placements of the lights 130 in a three-dimensionalperspective, with numerous of the lights being positioned on features ofthe interactive stage structure 150, such as on the frame 160, the posts162, or the walls 164.

In FIG. 2C, an interior surface 168 of the walls 164 is depicted. In anexample, the interior surface 168 may include a matte (non-smooth)fabric material designed to absorb light (infrared or visible) such thatlight is not reflected off the interior surface 168, because lightreflected of such surfaces may interfere with operation of the dualcamera system 300 or exposures of a produced video. In an example, theinterior chamber may include one or more user-interactive components,which may or may not be visible from within the camera field of view.These components may provide the output of video and information fromwithin an interior video display 180, such as a display screen mountedon one of the walls 164.

FIGS. 2D-2E illustrate interior views of the interactive stage structure150 for example scenarios of capturing video of a subject. FIG. 2Dprovides a view from an orientation of a human user (e.g., human user110) within the field of view of the camera system 300 and theteleprompter video display system 200. As depicted, the lights 130 arefocused to surround and aim visible light on the human user. As thehuman user faces forward, the human user will perceive the output of thevideo display from the teleprompter video display system 200. FIG. 2Eprovides a view from an opposite orientation, towards the human user(e.g., human user 110), from the orientation of the camera system 300and the teleprompter video display system 200.

Although numerous aspects of the frame 160, the posts 162, or the walls164 are depicted as being visible in FIGS. 2A-2E, in some examples,these structures may be wrapped, disguised, or hidden behind otherstructures or facades. Additional doors or openings not shown may beintegrated into the walls 164. Further, additional electronic orinteractive components and special effects devices (e.g., fog/smokemachines, speakers, lasers) may be supported by the frame 160, the posts162, the walls 164, or other aspects of the structure 150.

FIGS. 3A-3C illustrate front and perspective views of the dual camerasystem 300 within use of the teleprompter video display system 200.Specifically, as shown in FIG. 3A, the dual camera system 300 ispositioned on the rear platform 230, with the second camera unit 320 (avisible light camera) facing directly forward towards the opening 240.The hot mirror 350, as discussed above, filters out IR lightwavelengths, allowing the second camera unit 320 to capture video fromvisible lights. At the same time, the hot mirror 350 reflects the IRlight wavelengths towards the first camera unit 310 (an infrared lightcamera) that faces a direction perpendicular to the opening 240.

FIG. 3B further illustrates a front view of the teleprompter videodisplay system 200, with portions of the dual camera system 300 beingillustrated in FIG. 3B as visible through the two-way display screen 250through opening 240. The housing 210 of the teleprompter video displaysystem 200 may also define sides (e.g., a shroud) used to block lightcoming from other directions from reaching the two-way display screen250, and a display reflector 260 hosted within a frame 270. In anexample, the display reflector 260 outputs a reverse projection from thedisplay source (not shown) that is then reflected off the two-waydisplay screen 250; in another example not depicted, the display sourceis hosted by the frame 270 to directly output the reverse projectiontowards the two-way display screen 250.

FIG. 3C further illustrates a rear perspective view of the telepromptervideo display system 200 and the dual camera system 300 attachedthereon. The teleprompter video display system 200 may also include avariety of other structural and functional components not depicted forsimplicity.

In an example, the teleprompter video display system 200 may include amonitor to project light onto the two-way display screen 250, forexample reflected from the monitor by a mirror or a series of mirrors toreach the two-way display screen 250. The monitor may be located, forexample, below the dual camera system 300. In another example, a mirrormay be used to reflect light from a projector onto the two-way displayscreen 250. The projector may be located remotely from the telepromptervideo display system 200.

FIGS. 4A-4C illustrate top and perspective views of the dual camerasystem 300 with use in a video projection system 400. In an example, thevideo projection system 400 is provided as a replacement of theteleprompter video display system 200. The video projection system 400provides for use of a projection screen 440 in place of variousteleprompter arrangements and the two-way display screen 250. However,it will be apparent that many of the components and approaches utilizedin the teleprompter video display system 200 and the associatedplacement of the dual camera system 300 may also be applicable to theenvironment of the video projection system 400.

FIG. 4A illustrates a side perspective view of the video projectionsystem 400, which provides video output 420 from a projector unit 410towards a projector screen surface 430. In an example, the projectorscreen surface 430 is provided from a projection screen 440 made ofspecialized two-way glass (a reciprocal mirror). In an example, theprojection screen is made of a microlens array (MLA) material thatallows projected light (e.g., video output 420) to be reflected, whileallowing other light (e.g., visible and infrared light from the field ofview) to pass through to the camera. Other materials and forms oftwo-way projection screens may also be utilized.

FIG. 4B illustrates a top perspective view of the video projectionsystem 400, specifically showing the projection of the video output 420onto a first side 430A of the projection screen 440. FIG. 4C similarlyillustrates a rear perspective view of the video projection system 400,specifically showing the arrangement of the dual camera system 300relative to a second side 430B of the projection screen 440.

FIGS. 5A-5B illustrate side views of infrared and visible light sourcesfor use with the near-infrared video compositing camera system,according to an example. In FIG. 5A, the human user 110 is positioned inan environment 500A relative to visible light emitters (e.g., lights130) within a subject area between the teleprompter video display system200 and the backdrop 120. The field of view that can be captured by thedual camera system 300 (hosted on the teleprompter video display system200) includes part of the subject area, and is dependent based on thefocal distance, lens, and positioning of the dual camera system 300relative to any objects in the field of view (e.g., the human user 110).

FIG. 5B illustrates a path traveled by infrared light 125 from infraredlight emitters (e.g., an array of IR light emitters 124, suspended in agrid 126) and by visible light 135 from visible light emitters (e.g.,lights 130). In an example, the infrared light 125 travels from behind ahuman user 110 toward the dual camera system 300 (visible in FIG. 5A,not visible in FIG. 5B). As the infrared light 125 travels to the dualcamera system 300, part of the infrared light 125 is blocked by thehuman user 110 from reaching the dual camera system 300. Using theinfrared light that reaches the dual camera system 300, collected by aninfrared detection camera, a shape of the human user 110 can bedetermined. In an example, light that reaches the infrared detectioncamera represents a background area, which may be subtracted out of avisible light image taken by visible light detection camera of the dualcamera system 300 arranged to view the same field of view as theinfrared detection camera. The visible light 135 reflects off of andilluminates the human user 110. The reflected light reaches the visiblelight detection camera, which captures an image including the human user110.

FIG. 6 illustrates an example detailed view of video capture and outputfrom the near-infrared video compositing camera system with use of theteleprompter video display system 200. As shown, the human user 110, whois facing the teleprompter video display system 200 and the displayscreen 250, is able to view an output of a real-time video feed beingcaptured by the dual camera system. The display reflector 260 isarranged to project visible light originating from the display sourcetowards the display screen 250; the visible light is reflected by thedisplay screen to provide a video output from the display source that isvisible from the perspective of the user.

In a properly calibrated setting, where the user is not locatedimmediately next to the teleprompter video display system 200 (e.g., theuser is a suitable distance from the system 200), the reflection fromthe display screen 250 may provide a real-time output of acomputer-modified video including the user. In this fashion, the displayscreen 250 may serve as type of a video monitor for live preview andplayback of video applications. In performance environments, thestructure of the teleprompter video display system 200 (or, of the dualcamera system 300) may be disguised or hidden from the performance area,to provide a further illusion of a monitor rather than a camera source.

FIG. 7 illustrates an overview of a visible and near-infrared videoprocessing sequence 700 for video compositing, according to an example.The sequence 700 can start with infrared light permeating a visiblyblack (or dark, e.g., gray) backdrop 710. The infrared light can beemitted by a plurality of infrared LEDs. A first portion of the infraredlight is blocked by a human user 715 and a second portion of theinfrared light reaches a hot mirror 720. The hot mirror 720 is alignedto reflect infrared light to a near infrared camera 730, and permitsvisible light (e.g., reflected off the human user 715 from a lightingelement) to pass through the hot mirror 720 to reach a color camera 740(e.g., a RGB camera). The color camera creates an image 745 of the humanuser with a visible light background present. The near infrared camera730 creates an infrared image 735, which includes an illuminatedbackground portion and a darkened portion (e.g., silhouette) of thehuman user 715, corresponding to the second portion and the firstportion of infrared light, respectively.

The images 735 and 745 are sent to a video capture card 750, which canstore the images (frames) of the video capture. A software virtualeffects (VFX) system 755 can be used to further process the images. Forexample, a color camera feed 760 (e.g., including image 745) can becombined with a NIR camera feed 765 (e.g., including image 735) tocreate a luma matte 780. Further processing, such as color correction770 on the color camera feed 760 or thresholding 775 on the NIR camerafeed 765 may be performed by the software VFX system 755.

In an example, information from the thresholding 775 may be used toidentify a first portion of the image 735 that is foreground and asecond portion of the image 735 that is background. Because the colorcamera 740 and the near infrared camera 730 are focused on the same (orsubstantially the same) field of view, overlaying the foreground andbackground portions on the image 745, allows for background subtractionof the image 745 using the luma matte 780 to create a color image withalpha channel 785. For example, a portion of the image 745 correspondingto locations of the second portion of the image 735 that is thebackground can be removed from the image 745 to create the image 785,while leaving a portion of the image 745 corresponding to locations ofthe first portion of the image 735 that is the foreground. After thebackground is removed from image 785, a virtual background (or otherbackground image or video frame) can be added using the software VFXsystem 755.

FIG. 8 illustrates a flowchart 800 of an example technique forgenerating a near-infrared matte and performing video compositing withthe matte via an electronic video processing system. The technique ofthe flowchart 800 may be performed by any of the components, logic, orsystems described herein. Further, the order and type of the operationsdepicted in the flowchart 800 may be added, modified, or substitutedusing any of the operations or functions described herein. Thus,although the flowchart 800 and the following operations are depictedfrom the perspective of a video processing system, other types ofoperations, systems, and devices may be used to perform these or similaroperations.

In an example, the method of the flowchart 800 begins with the controlof illumination of subject area with visible and infrared light(operation 810). In some examples, this illumination is pre-calibratedto particular characteristics of the subject area and surroundingstructure. In other examples, this illumination is varied and changesdepending on characteristics of the objects or humans within the subjectarea or camera field of view. Various forms of automatic and manualadjustments of lighting (e.g., to match a particular scenario) may alsobe incorporated.

The method of the flowchart 800 continues with the obtaining (e.g.,capturing, downloading, accessing) of an infrared video stream of asubject area, originating from an infrared camera (operation 820), andthe obtaining (e.g., capturing, downloading, accessing) of an RGB videostream of the subject area, originating from a visible light camera(operation 830). Based on these video streams, further processing,generation of a matte, and compositing may be performed.

In an example, the video streams are captured in software of a computingsystem (a video processing system) using one or more video capturecards. The digital capture of the video within a video processing systemenables the ability to digitally composite and process the video sourceswith backgrounds, foregrounds, fluid dynamics simulations, computervision data sources, face tracking algorithms, and other aspects ofadjustments and processing. As one specific example of furtherprocessing, various adjustments such as thresholding and colorcorrection (operation 840) may be implemented on the RGB or infraredvideo streams.

The method of the flowchart 800 continues with a generation of aforeground matte from a RGB video stream (operation 850), based on asilhouette of any objects (human and non-human) captured in the infraredvideo stream. The techniques discussed above with reference to FIG. 7may be used to establish the foreground matte, to produce a series ofcolor images or frames (defining the foreground matte) having respectivealpha channels (defining the transparency). A new background, to replacethe alpha channel, may be obtained (e.g., captured, downloaded,accessed) (operation 860).

The method of the flowchart 800 continues with the identification andcontrol of visual effects for the foreground matte or the backgroundvideo content (operation 870), and the application of such visualeffects (operation 880). These visual effects may be user-selected,automatically selected, implemented based on a predefined script orscenario, or the like. These visual effects may include graphicalchanges to the video (e.g., the addition or changes of colors, additionof computer-generated graphics) or the playback parameters of the video(e.g., to apply slow-motion or fast-motion playback effects). Finally,the composited video output may be provided (e.g., produced, generated,exported, etc.) (operation 890). In further examples, additional visualeffects may be applied after the foreground matte and background iscomposited or otherwise combined into a video stream. Accordingly,real-time video manipulation and effects may be incorporated into anoutput of the composited video.

As discussed in the various examples herein, the composited video outputmay be provided to a real-time display after the application of thevisual effects (e.g., for output via systems 200 or 400). In stillfurther examples, the video processing system may dynamically record aperformance of a user or set of users captured within the environment,and allow digital downloads or delivery of recorded video via anelectronic medium or network.

FIG. 9 illustrates a flowchart 900 of an example method for videocompositing in an immersive video environment based on infrared video.In a similar manner as flowchart 800, the order and type of theoperations depicted in the flowchart 900 may be added, modified, orsubstituted using any of the operations or functions described herein.

The method of the flowchart 900 begins with the capturing, at aninfrared detection camera of a matte camera system, of a first portionof infrared light originating from an infrared light source (operation910), relative to the performance area. In an example, this infraredlight is captured through a display screen arranged between the mattecamera system and the infrared light source; in a further example, theinfrared light is projected from behind the performance area, whichcauses a second portion of the infrared light to be blocked by anyobjects (including humans) in the performance area. In an example, theperformance area is hosted within a structure arranged to host thedisplay source, the camera system, and the visible light source; in afurther example, the structure is arranged such that visible light fromoutside the structure is prevented from reaching the visible lightdetection camera of the camera system.

The method of the flowchart 900 continues with the capturing, at avisible light detection camera of the matte camera system, of visiblelight originating from a visible light source and traveling through thedisplay screen (operation 920). As discussed in the configurationsherein, the visible light originating from the visible light source maybe received at the display screen at an angle to a face of the displayscreen. In an example, the visible light detection camera produces RGBvideo of a human subject in the performance area, for a human subjectwho is located between the infrared light source and the display screenwithin the performance area. In a further example, the visible lightsource is arranged such that the visible light originating from thevisible light source reflects off the human subject to be detected bythe visible light detection camera.

The method of the flowchart 900 continues, with the determination of abackground portion of video for the captured visible light, based on theinfrared light that is blocked by object(s) in the performance area(operation 930). In an example, a second portion of the infrared light(that originates from the infrared light source) from behind is used toestablish a silhouette, which identifies a background portion and aforeground portion of the real-time video display. This is followed bythe processing operations in the flowchart 900, of the removal of theidentified background portion from the captured visible light videostream (operation 940).

The method of the flowchart 900 continues, with the compositing of thevideo stream produced from the visible light stream (having a backgroundremoved) with other background video content (operation 950). Assuggested herein, the background video content may be a virtualbackground (e.g., a computer-generated graphic, a photo, a video frameor set of video frames, or the like). In an optional example, variousvisual effects may be applied to the composited video stream (operation960), including to either or both of foreground or background of thecomposited video.

The method of the flowchart 900 concludes with the output of thecomposited video stream, such as with a projection onto one or moredisplay screens in the performance area (operation 970). In an optionalexample, this may be accompanied by various sound, light, and otherspecial effects in the performance area based on the composited videostream (operation 980). In an example, the monitor and the matte camerasystem may be housed by a teleprompter housing, such that theteleprompter housing is positioned to allow viewing by a human locatedwithin a defined environment (e.g., within an enclosed structure). Inanother example, the display source is a visible light projector, andthe visible light projector is adapted to directly or indirectly projectthe real-time video onto the display screen, such as in a scenario wherethe visible light projector is arranged between the infrared lightsource and the display screen to allow viewing by a human within thedefined environment (e.g., within an enclosed structure). In a furtherexample, the projection screen is a microlens array (MLA) displayscreen.

In an example, the output of the video includes projecting light from adisplay source onto the display screen, such that the projection of thelight onto the display screen is visible to a human subject in aperformance area. Other variations of displays and display arrangementsthat allow capture and presentation of video from a common location, inreal-time, may also be utilized in addition or in substitute to thesearrangements.

FIG. 10 illustrates a block diagram of example processing components andsubsystems operable for video compositing based on NIR video, based onthe techniques described herein. For example, a series of systems,including a video input/output system 1010, a video processing system1020, and a video capture system 1030, may be operably coupled anddeployed in a video capture environment such as with the environmentsdepicted in FIGS. 1 to 6 and with the techniques described in FIGS. 7 to9. For instance, the video capture system 1030 may embody features ofthe dual camera system 300; the video input/output system 1010 mayembody features of the teleprompter video display system 200 or thevideo projection system 400; the video processing system may embody acomputer system adapted to perform or control the functionality of theflowcharts 700, 800, 900.

The video input/output system 1010, the video processing system 1020,and the video capture system 1030 may include respective hardwarecomponents, such as processing circuitry 1011, 1021, 1031 to executeinstructions, memory 1012, 1022, 1032 used with the processing circuitryto execute instructions and provide data, data storage 1013, 1023, 1033to host and store instructions and data, and networking circuitry 1014,1024, 1034 to communicate (e.g., receive and transmit) data amongsystems via a network. The hardware components may operate with use offurther hardware and software-implemented components (not depicted)located among the systems 1010, 1020, 1030 for user input, output, andprocessing, such as aspects of a graphical user interface, outputdevices (such as to provide output of the graphical user interface) andinput devices (such as to provide input for processing and actions inthe graphical user interface).

In an example, the video input/output system 1010 is a controllablesystem configured to implement data viewing and capture operations forobtaining NIR matte and formatted video data. For instance, in thecontext of the immersive video environment described herein (e.g., asdepicted in FIGS. 2A-2E), the video input/output system may controlnumerous electrical components (e.g., lights, video screens, audiooutputs, environment special effects). As referenced throughout thisdisclosure, the configuration and use of the immersive video environmentmay be adapted according to the human users or objects within theenvironment, the selected video or image backgrounds, the selectedspecial effects in the environment or for the video or image, and thelike. Thus, it will be understood that the control and performancecapabilities of the video input/output system 1010 may vary depending onthe deployed performance environment, the controllable devices withinsuch environment, and the use of the controllable devices and settingswithin such environment.

In an example, the video input/output system 1010 may include a displaydevice 1016 for outputting real-time video and video special effects(e.g., produced from the video processing system 1020), and a userinterface 1017 for providing inputs for control or changes of thereal-time video and the video special effects (e.g., to effect controlof features of the video processing system 1020 or the video capturesystem 1030). The video input/output system 1010 also may includecomponents (e.g., programmed or specially arranged circuitry) forimplementing environmental control features, such as through: lightingcontrol functionality 1018 that implements and executes lightingscenarios and settings among lighting devices (e.g., by controllinglights 124, 130); effects control functionality 1019 that implements andexecutes effect outputs in the environment (e.g., by controllingconnected video, audio, or special effects devices within theenvironment of the interactive stage structure 150). In an example,aspects of the functionality 1018, 1019 may be scripted or automated toimplement automatic settings for particular video use cases. In afurther example, an output device and an input device (not depicted) areused to engage the user interface 1017 with use of the processingcircuitry 1011 and memory 1012, to implement user-defined settings forfeatures of the lighting control functionality 1018 and the effectscontrol functionality 1019.

In addition to previously described features and functionality, thevideo processing system 1020 is depicted as compositing functionality1026 and effects functionality 1027. In an example, the compositingfunctionality 1026 is adapted to process camera video streams (e.g.,camera feeds 760, 765) from a NIR/Visible camera system (e.g., dualcamera system 300), and create a matte (e.g., luma matte 780) andgenerate output image and video (e.g., image with alpha channel 785)from the two respective video streams. The effects functionality 1027 isalso adapted to implement post-processing video effects on all or aportion of the video streams (e.g., with the addition of additionalvideo objects or layers, the distortion of colors, shapes, orperspectives in the video, and any other number of other video changes).In a further example, the video processing system 1020 may operate as aserver, to receive and process video data obtained from the videocapture system 1030, and to serve video data output to the videoinput/output system 1010.

In addition to previously described features and functionality, thevideo capture system 1030 may include components of a dual camerasystem, such as a near-infrared camera 1036 and a visible light camera1037. In an example, the near-infrared camera 1036 includes a sensor todetect NIR light (e.g., emitted in the interactive stage structure 150environment from IR light emitters 124) and produce a NIR video stream,while the visible light camera includes a sensor to detect visible light(e.g., emitted in the interactive stage structure 150 environment from avisible light source such as lights 130) and produce a RGB video stream.The respective video streams are then communicated to the videoprocessing system 1020 for compositing and video effects. In furtherexamples, functionality (not depicted) may provide pre-processing andadjustments of the video stream(s) before communication to the videoprocessing system 1020 or the video input/output system 1030. Further,raw or pre-processed captured video data may be communicated to thevideo processing system 1020 and the video input/output system 1010 inreal time, in a delayed fashion, or upon demand.

In an example, the features of the various systems 1010, 1020, 1030 maybe integrated or combined into a single system, device, or sub-system.In other examples, the features of the various systems 1010, 1020, 1030may be distributed among multiple computing machines, including inscenarios involving the use of external (e.g., remote,network-connected) video processing systems. Other variations toimplement the video compositing and effects may be implemented byadditional hardware provided within the systems 1010, 1020, 1030, and anaccompanying use environment (e.g., within interactive stage structure150).

The components, methods, applications, and so forth described inconjunction with FIGS. 7-9 (and deployed in the examples set forth forFIGS. 1-6 and 10) are implemented in some embodiments in the context ofa machine and an associated software architecture for video processing.The paragraphs below describe representative software architecture(s)and machine (e.g., hardware) architecture(s) suitable for use with thedisclosed embodiments. For example, software architectures may be usedin conjunction with hardware architectures to create devices andmachines tailored to particular purposes. For example, a hardwarearchitecture coupled with a software architecture may create a videoprocessing device or like graphical output device. Not all combinationsof such software and hardware architectures are presented here, as thoseof skill in the art can readily understand how to implement thedisclosed subject matter in different contexts from the disclosurecontained herein.

FIG. 11 is a block diagram illustrating components of a machine 1100,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 11 shows a diagrammatic representation of the machine1100 in the example form of a computer system, within which instructions1116 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 1100 to perform any oneor more of the methodologies discussed herein may be executed. Theinstructions 1116 transform the machine into a machine programmed tocarry out the described and illustrated functions in the mannerdescribed. In alternative embodiments, the machine 1100 operates as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 1100 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1100 may comprise, but not be limitedto, a server computer, a client computer, PC, a tablet PC, a hybridtablet, a laptop computer, a netbook, a personal digital assistant(PDA), an entertainment media system, a cellular telephone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smarthome device (e.g., a smart appliance), other smart devices, a webappliance, a network router, a network switch, a network bridge, or anymachine capable of executing the instructions 1116, sequentially orotherwise, that specify actions to be taken by the machine 1100.Further, while only a single machine 1100 is illustrated, the term“machine” shall also be taken to include a collection of machines 1100that individually or jointly execute the instructions 1116 to performany one or more of the methodologies discussed herein.

The machine 1100 may include processors 1110, memory/storage 1130, andI/O components 1150, which may be configured to communicate with eachother such as via a bus 1102. In an example, the processors 1110 (e.g.,a Central Processing Unit (CPU), a Reduced Instruction Set Computing(RISC) processor, a Complex Instruction Set Computing (CISC) processor,a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), anASIC, a Radio-Frequency Integrated Circuit (RFIC), another processor, orany suitable combination thereof) may include, for example, a processor1112 and a processor 1114 that may execute the instructions 1116. In anexample, a processor includes multi-core processors that may comprisetwo or more independent processors (sometimes referred to as “cores”)that may execute instructions contemporaneously. Although FIG. 11 showsmultiple processors 1110, the machine 1100 may include a singleprocessor with a single core, a single processor with multiple cores(e.g., a multi-core processor), multiple processors with a single core,multiple processors with multiples cores, or any combination thereof.

The memory/storage 1130 may include a memory 1132, such as a mainmemory, or other memory storage, and a storage unit 1136, bothaccessible to the processors 1110 such as via the bus 1102. The storageunit 1136 and memory 1132 store the instructions 1116 embodying any oneor more of the methodologies or functions described herein. Theinstructions 1116 may also reside, completely or partially, within thememory 1132, within the storage unit 1136, within at least one of theprocessors 1110 (e.g., within the processor's cache memory), or anysuitable combination thereof, during execution thereof by the machine1100. Accordingly, the memory 1132, the storage unit 1136, and thememory of the processors 1110 are examples of machine-readable media.

A machine-readable medium includes a device able to store instructions(e.g., instructions 1116) and data temporarily or permanently and mayinclude, but is not limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, optical media, magneticmedia, cache memory, other types of storage (e.g., Erasable ProgrammableRead-Only Memory (EEPROM)), and/or any suitable combination thereof.Thus, a machine-readable medium may include a single medium or multiplemedia (e.g., a centralized or distributed database, or associated cachesand servers) able to store the instructions 1116. A machine-readablemedium may also include medium, or combination of multiple media, thatis capable of storing instructions (e.g., instructions 1116, stored in anon-transitory manner) for execution by a machine (e.g., machine 1100),such that the instructions, when executed by one or more processors ofthe machine (e.g., processors 1110), cause the machine to perform anyone or more of the methodologies described herein. Accordingly, amachine-readable medium refers to a single storage apparatus or device,as well as “cloud-based” storage systems or storage networks thatinclude multiple storage apparatus or devices.

The I/O components 1150 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and the like. The specificI/O components 1150 that are included in a particular machine willdepend on the type of machine. For example, portable machines such asmobile phones may include a touch input device or other such inputmechanisms, while a headless server machine will likely not include sucha touch input device. It will be understood that the I/O components 1150may include many other components that are not shown in FIG. 11. The I/Ocomponents 1150 are grouped according to functionality merely forsimplifying the following discussion, as the illustrated grouping is notintended to be limiting. Various components of the following I/Ocomponents 1150 may be used, for example, in connection with bot-humaninteraction features in connection with the bots discussed herein.

In various examples, the I/O components 1150 may include outputcomponents 1152 and input components 1154. The output components 1152may include visual components (e.g., a display such as a plasma displaypanel (PDP), a light emitting diode (LED) display, a liquid crystaldisplay (LCD), a projector, or a cathode ray tube (CRT)), acousticcomponents (e.g., speakers), haptic components (e.g., a vibratory motor,resistance mechanisms), other signal generators, and so forth. The inputcomponents 1154 may include alphanumeric input components (e.g., akeyboard, a touch screen configured to receive alphanumeric input, aphoto-optical keyboard, or other alphanumeric input components), pointbased input components (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or another pointing instrument), tactileinput components (e.g., a physical button, a touch screen that provideslocation and/or force of touches or touch gestures, or other tactileinput components), audio input components (e.g., a microphone), and thelike.

In further example embodiments, the I/O components 1150 may includebiometric components 1156, motion components 1158, environmentalcomponents 1160, or position components 1162, among a wide array ofother components. For example, the biometric components 1156 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebio signals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), measure exercise-related metrics (e.g.,distance moved, speed of movement, or time spent exercising) identify aperson (e.g., voice identification, retinal identification, facialidentification, fingerprint identification, or electroencephalogrambased identification), and the like. The motion components 1158 mayinclude acceleration sensor components (e.g., accelerometer),gravitation sensor components, rotation sensor components (e.g.,gyroscope), and so forth. The environmental components 1160 may include,for example, illumination sensor components (e.g., photometer),temperature sensor components (e.g., one or more thermometers thatdetect ambient temperature), humidity sensor components, pressure sensorcomponents (e.g., barometer), acoustic sensor components (e.g., one ormore microphones that detect background noise), proximity sensorcomponents (e.g., infrared sensors that detect nearby objects), gassensors (e.g., gas detection sensors to detect concentrations ofhazardous gases for safety or to measure pollutants in the atmosphere),or other components that may provide indications, measurements, orsignals corresponding to a surrounding physical environment. Theposition components 1162 may include location sensor components (e.g., aGlobal Position System (GPS) receiver component), altitude sensorcomponents (e.g., altimeters or barometers that detect air pressure fromwhich altitude may be derived), orientation sensor components (e.g.,magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1150 may include communication components 1164operable to couple the machine 1100 to a network 1180 or devices 1170via a coupling 1182 and a coupling 1172, respectively. For example, thecommunication components 1164 may include a network interface componentor other suitable device to interface with the network 1180. In furtherexamples, the communication components 1164 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 1170 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 1164 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1164 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components, or acoustic detection components (e.g.,microphones to identify tagged audio signals). In addition, a variety ofinformation may be derived via the communication components 1164, suchas location via Internet Protocol (IP) geolocation, location via Wi-Fi®signal triangulation, location via detecting an NFC beacon signal thatmay indicate a particular location, and so forth.

In various examples, one or more portions of the network 1180 may be anad hoc network, an intranet, an extranet, a virtual private network(VPN), a local area network (LAN), a wireless LAN (WLAN), a WAN, awireless WAN (WWAN), a metropolitan area network (MAN), the Internet, aportion of the Internet, a portion of the Public Switched TelephoneNetwork (PSTN), a plain old telephone service (POTS) network, a cellulartelephone network, a wireless network, a IEEE 802.11 Wi-Fi® network,another type of network, or a combination of two or more such networks.For example, the network 1180 or a portion of the network 1180 mayinclude a wireless or cellular network and the coupling 1182 may be aCode Division Multiple Access (CDMA) connection, a Global System forMobile communications (GSM) connection, third Generation PartnershipProject (3GPP) connection such as via a fourth generation (4G) or fifthgeneration (5G) wireless network, or another type of cellular orwireless coupling. In this example, the coupling 1182 may implement anyof a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, UniversalMobile Telecommunications System (UMTS), High Speed Packet Access(HSPA), Long Term Evolution/Long Term Evolution-Advanced (LTE/LTE-A),Worldwide Interoperability for Microwave Access (WiMAX), includingstandards of such protocols, communication protocols defined by variousstandard-setting organizations, other long range protocols, or otherdata transfer technology.

The instructions 1116 may be transmitted or received over the network1180 using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components1164) and utilizing any one of a number of well-known transfer protocols(e.g., HTTP). Similarly, the instructions 1116 may be transmitted orreceived using a transmission medium via the coupling 1172 (e.g., apeer-to-peer coupling) to the devices 1170. Thus, a transmission mediummay include any intangible medium that is capable of storing, encoding,or carrying the instructions 1116 for execution by the machine 1100, andincludes digital or analog communications signals or other intangiblemedia to facilitate communication of such software.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with others. Otherembodiments may be used, such as by one of ordinary skill in the artupon reviewing the above description. In the above Detailed Description,various features may be grouped together to streamline the disclosure.However, the claims may not set forth every feature disclosed herein asembodiments may feature a subset of said features. Further, embodimentsmay include fewer features than those disclosed in a particular example.Thus, the following claims are hereby incorporated into the DetailedDescription, with a claim standing on its own as a separate embodiment.The scope of the embodiments disclosed herein is to be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A system comprising: a camera system including avisible light detection camera and an infrared detection camera, thecameras arranged to capture video from a single capture point to image aperformance area; a visible light source arranged to emit visible lightonto the performance area, the visible light to be reflected from theperformance area towards the camera system; an infrared light sourcearranged to emit infrared light onto the performance area, the infraredlight to be emitted through the performance area towards the camerasystem; a display source to provide video output; and a display screenarranged between the camera system and the infrared light source,wherein the display screen is arranged to cause a space for theperformance area to be defined between the display screen and theinfrared light source, and wherein the display screen is arranged to beviewable within the performance area while a first portion of infraredlight originating from the infrared light source is blocked fromreaching the infrared detection camera when a human subject is locatedwithin the performance area; wherein the display screen is furtherarranged to: permit a second portion of the infrared light originatingfrom the infrared light source to pass through the display screen to theinfrared detection camera; permit the visible light originating from thevisible light source and reflected from the performance area to passthrough the display screen to the visible light detection camera; andreflect visible light originating from the display source towards theperformance area, wherein the visible light originating from the displaysource is reflected by the display screen to provide the video outputfrom the display source that is visible in the performance area.
 2. Thesystem of claim 1, wherein the visible light detection camera producesRGB video of the human subject in the performance area, for the humansubject located between the infrared light source and the display screenwithin the performance area, and wherein the visible light source isarranged such that the visible light originating from the visible lightsource reflects off the human subject to be detected by the visiblelight detection camera.
 3. The system of claim 1, further comprising abackdrop, the backdrop including a material that permits the infraredlight to pass through the backdrop while blocking visible light.
 4. Thesystem of claim 3, wherein the backdrop is arranged between the infraredlight source and the performance area, such that infrared light from theinfrared light source passes through the backdrop, wherein the firstportion of infrared light is blocked by the human subject, and whereinthe second portion of the infrared light is not blocked by the humansubject and passes through the display screen to be detected by theinfrared detection camera.
 5. The system of claim 1, wherein theinfrared light source includes a plurality of parallel infrared lightbars spaced vertically up from a floor of the performance area.
 6. Thesystem of claim 1, further comprising a structure surrounding theperformance area, the structure arranged to host the display source, thecamera system, and the visible light source, and wherein the structureis further arranged such that visible light from outside the structureis prevented from reaching the visible light detection camera of thecamera system.
 7. The system of claim 1, wherein the visible lightoriginating from the display source and reflected by the display screenprovides a real-time video display of the human subject superimposed ona virtual background.
 8. The system of claim 7, wherein blocking of thesecond portion of the infrared light originating from the infrared lightsource is used to identify a background portion and a foreground portionof the real-time video display, wherein the background portion of thereal-time video display is removed as the human subject is superimposedon the virtual background.
 9. The system of claim 1, wherein the displaysource is a monitor, wherein the monitor is arranged to project thevisible light originating from the display source onto the displayscreen, and wherein the monitor and the camera system are hosted by ateleprompter housing.
 10. The system of claim 1, wherein the displaysource is a projector, wherein the projector is arranged to project thevisible light onto the display screen, and wherein the projector isarranged between the infrared light source and the display screen. 11.The system of claim 10, wherein the display screen is a microlens arraydisplay screen.
 12. The system of claim 1, wherein the visible lightoriginating from the visible light source is received at the displayscreen at an angle to a face of the display screen.
 13. The system ofclaim 1, further comprising: an audio output device, wherein the audiooutput device is arranged to output sound to the performance area, andwherein the sound corresponds to the video output from the displaysource.
 14. A method, comprising: capturing, at an infrared detectioncamera of a matte camera system, a first portion of infrared lightoriginating from an infrared light source through a display screenarranged between the matte camera system and the infrared light source;capturing, at a visible light detection camera of the matte camerasystem, visible light originating from a visible light source andtraveling through the display screen, wherein the matte camera system isarranged to capture video for the infrared detection camera and thevisible light detection camera from a single capture point into aperformance area; and projecting, from a display source, light onto thedisplay screen, wherein the projection of the light onto the displayscreen is visible to a human subject in the performance area, whereinthe display screen is arranged to cause a space for a performance areato be defined between the display screen and the infrared light source;wherein a second portion of infrared light originating from the infraredlight source is blocked from reaching the infrared detection camera bythe human subject in the performance area when the human subject islocated within the performance area, wherein the infrared light isemitted through the performance area towards the matte camera system.15. The method of claim 14, wherein the display source is a monitor,wherein the monitor is arranged to project light onto the displayscreen, and wherein the monitor and the matte camera system are housedwithin a teleprompter housing.
 16. The method of claim 14, wherein thedisplay source is a projector, wherein the projector is arranged toproject the light onto the display screen, and wherein the projector isfurther arranged between the infrared light source and the displayscreen.
 17. A method, comprising: capturing, at an infrared detectioncamera of a matte camera system, an infrared video stream of a firstportion of infrared light originating from an infrared light source,wherein the infrared light source transmits the first portion ofinfrared light to the infrared detection camera through a display screenarranged between the matte camera system and the infrared light source,and wherein the infrared light source emits the infrared light within anenclosed structure; capturing, at a visible light detection camera ofthe matte camera system, a visible video stream of visible lightoriginating from a visible light source, wherein the visible lightsource emits the visible light within the enclosed structure, andwherein the visible light is received through the display screen,wherein the matte camera system is arranged to capture video for theinfrared detection camera and the visible light detection camera from asingle capture point into a performance area located within the enclosedstructure; determining a background portion of the visible video streamcorresponding to an area surrounding the first portion of infrared lightin the infrared video stream, wherein the area surrounding the firstportion of infrared light corresponds to a second portion of theinfrared light that is blocked by an object within the performance areaof the enclosed structure as infrared light is emitted through theperformance area towards the matte camera system, and wherein thedisplay screen is arranged to cause a space for the performance area tobe defined between the display screen and the infrared light source;removing the background portion of the visible video stream; generatinga real-time video including a real-time video capture of the object fromthe visible video stream and a virtual background replacing the removedbackground portion; and projecting, from a display source, the real-timevideo reflected off the display screen, wherein the real-time video isvisible within the enclosed structure from a perspective of the object.18. The method of claim 17, wherein the display source is a monitor,wherein the monitor is adapted to project visible light representing thereal-time video onto the display screen, wherein the monitor and thematte camera system are housed by a teleprompter housing, and whereinthe teleprompter housing is positioned within the enclosed structure toallow viewing by a human located within the enclosed structure.
 19. Themethod of claim 17, wherein the display source is a visible lightprojector, the visible light projector to project the real-time videoonto the display screen, wherein the visible light projector is arrangedbetween the infrared light source and the display screen, and whereinthe display screen is positioned within the enclosed structure to allowviewing by a human located within the performance area of the enclosedstructure.
 20. The method of claim 14, wherein the infrared light sourceincludes a plurality of parallel infrared light bars spaced verticallyup from a floor of the performance area.